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ABSTRACT 

Ideas on the necessity for evaluating alternative 
programs and on developing evaluation programs are examined in this 
position paper* Many alternative schools, opposed to measurement of 
their program frcm a philosophical and defensive standpoint, view 
evaluation in a negative light because the purpose is not understood. 
However, evaluation can be an integral part of an ongoing formative 
evaluation process; establish credibility for an innovative program; 
identify workable educational strategies; and set the stage for 
student evaluation. Problems and issues in alternative education are 
variedo One of the problems menticaed is that the stereotype of a 
good evaluarion is one where no negative information is brought forth 
— whereas, in contrast, a good evaluation provides direction for 
program improvement. Another problem is that an Inadequacy of 
evaluation instru&ients md evaluators exists, other issues and 
problems relate to stringent demands, the role of behavioral 
objectives, and evaluation by external sources. In summary, 
evaluation needs to be an integral part of the planning process with 
the staff and the central administration establishing goals and 
objectives idiich can be evaluated. (SJM) 
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EVALUATING EDUCATIONAL ALTERNATIVES 



by Mike Hickey 



Introduction 

The "state ore the art" in evaluating alternative education programs can best 
be summarized in two statements: (1) an art it is not; and (2) the only 
state it is in is a state of general chaos. 

Given this somewhat negative overture, members of the task group on evalua- 
tion have addressed themselves to five main concerns: 

1. some reason for the state of evaluation 

2. purposes of. evaluation in alternative programs 

3. problems and Issues of evaluating alternatives 

4. consideration of some evaluative processes for alternatives 

5. roles for the Consortium in these evaluative efforts. 

It should be noted at the outset that this dicsussion focuses on program 
evaluation. This focus is necessitated by time limitations for the work 
group sessions in this conference and is not meant to imply any particular 
priority. Keither does it fall to recognise the interrelationship of pro- 
gram evaluation with evaluation of students and staff. 
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"State of the Art' ' 

At present, many individuals who are immersed in alternative education pro- 
grams are resistant to evaluation of their program, particularly by an out- 
side person or grcup. Tuis resistance is cere than just paranoia, although 
there is some of that, too. This resistance is based on an honest belief 
that evaluation is philosophically antithetical to the freedom and lack of 
overbearing structure upon which many alternatives are based. 

Another factor contributory to this situation is that of the i.t:£?ticaary de- 
fensiveness of some individuals in some programs. This defensiveness , often 
characterized by a demand to be left alone, again is sometimes justified 
when one considers the tenuous existence of most alternative programs. 

This defensiveness referred to above may stem in part from the abuse of 
evaluation through its application as a negative force, e.g., "proving" that 
a program is not effective. There is never justification for such an ap- 
proach, but that it does occur can be readily verified by scanning the brief 
literature of the alternative school movement. Tests were devexoped to mea- 
sure, not to classify, but the distortions of their purpose in order to 
group, to track, to categorize children have occurred with alarming frequen- 
cy. 

Beyond such flagrant misuses, however, much of the hostility toward evalua- 
tion has arisen from failure to understand what evaluation is all about. 
The blame for this failure rests equally with both sides: the evaluator for 
not clarifying what he is doing and why, and for imposing an evaluation 
design on the program; the program director for not requiring full explanation 
of the process and its underlying rationale and for not taking the initiative 
developing, at least partially, the evaluation design. 
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The foregoing factors then— philosophical opposition, defensiveness, nega- 
tive use of evaluation, and failure to define or to understand the purpose 
of evaluation — all contribute at least partially to the current negative 
aura which pervades evaluation in alternative education today. All of 
these concerns are valid, but the answer lies in eliminating their basis, 
not in rejecting evaluation. For alternative education programs, at least 
for those in public education, evaluation is part of today's reality, tlie 
price to be paid for spending the public's money. Alternatives are compat- 
ible with evaluation, especially if wa sieze the initiative and develop 
ways of improving the process and using it for the purposes of alternative 
education. The public ip demaixding accountability of its education pro- 
grams and accountability means, in part, evaJ.uation. If you still can't 
buy it that's fine, but ycu'd better find yourself another sugar daddy. 

Purposes of Evaluation in Mtemativcs 



Evaluation is essential in alternative prograips for at least four specific 
reasons. However, at a more general philosophical level the case might be 
stated as the maximi, "anything worth doing is worth doing well." This is 
especially true where the lives of chixdren and their future in our society 
are concerned. It can be assumed thac any program in existence was developed 
to meet a need. Evaluation can provide some indication of the extent to 
which that need is being met. To reject these indicators and settle for the 
aroma of success is intellectuciliy a cop out. 

The specific purposes of evaluation in alternative programs are at least four. 

First, and ;^erhaps of highest priority, is the purpose of internal self- 
improvement for the progrm, which in turn relates to the ongoing planning 
process (informal though it nay be) . 

Second, as a basis for establishing the credibility of the altemati/e pro- 
gram, evaluation must meet the demands of a variety of "publics." Like it or 
not, the regular program has established its credibility through the process 
of historical endurance during easier times. Alternative education must be 
prepared for assaults on its integrity because (a) it is a change, (b) it 
implies some weakness or lack in the regular program and (c) it diverts funds 
from the regular program. Tae alternatives — for awhile at least—will 
continuously be called upon to defend their honor through evaluation. 
Whether or not the program evaluates, other people do. 

Third, a primary rationale for the existence of alternatives within public 
education is that they become the means or the process by which public edu- 
cation evolves. Realistically, some strategies for educational alternatives 
will not work. Evaluation provides a base for identifying those that work 
and those that don't. From those that don't, needed changes in the alterna- 
tive program can be made. From those that do work, lessons can be learned 
which will eventually change the regular system. 

Finally, the evaluation of student progress is difficult without an adequate 
understanding of where the program itself stands. 

There are other reasons for evaluation of alternative education programs, but 
Q these four stand out as being of high priority. 

ERIC 
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ProbleiEs and Issues 

The\ following discussion of problems and issuas in alternative education 
covers a wide variety, but by no means exhausts the issues. Likewise, the 
order of discussion is not intended to connote any priority. For purposes 
of brevity, each is identified and somewhat defined, but no attempt is made 
in this working paper to estabUsh or discuss all the ramifications of these 
salient issues. 



. The evaluation process has unfortunately been stereotyped as one 
in which a "good" evaluation develops no negative information. 
This should not be the case. Negative evaluation data may be 
more productive in terms of the direction for program change they 
indicate, as opposed to positive data which often masks non-pro- 
ductive (but non-negative) program elements, A good evaluation, 
then, is one which provides information and direction for program 
improvement. 



. The inadequacy of many evaluation instruments—especially within 
the affective domain— is well known. However, all too often this 
fact is uoed as a rationalization for not evaluating. In altema- 
^^tiye education particularly the claim is frequently heard that 
"we're doing something different. It can't be evaluated like an 
ordinary school program." While there is some justification .to 
claims of weakness in many atfectivj measurement instruments and 
techniques, substantial progress has been made and it behooves us 
in alternative education to keep abreast of these developments. 
Excellent examples of these efforts include: 



a. the evaluation design for the METRO High School Program, 
Qiicago (Center for New Schools, 431 South Dearborn, Suite 
1527, Chicago 60605) 

b. the Ford Foundation's evaluation of Philadelphia's Parkway 
Program (Cambridge: OSTI) , 

c. THEORY INTO PRACTICE, Ohio State University, College of Edu- 
cation Journal. The April, 1969, issue focused on research 
in affective education. 



3. The '^ard data syndrome" is another rationalization for not eval- 
uating, or for misusing evaluation. It equates evaluation with 
standardized tests and discounts the value of "soft" data. In edu- 
cation, particularly alternative programs, it muse be recognized 
that a variety of measurement techniques are required. VJhile the 
precision and nature of the data may vary considerably, each plays 
an essential part in the process. We must accept the fact that in- 
direct measures become as important to evaluation and program im- 
provement as conventional direct ones. 

4. Frequently, the evaluation demands placed on alternative programs 
are far more stringent than any within the regular program. Like 
it or not, the fact is that the regular program has established 
its credibility through its historical endurance, while alternative 

^ programs are, £er sg^ suspect. Although inherently unfair, this 
en ir^" tendency may in the long run be to the ultimate benefit of alterna- 
jj^^^ tive education in that we are developing evaluation concurrently 
with program development. 
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5. Along with accountability has come the era of behavioral objectives 
(or b.o., as some refer to it). The constant state of flux that is 
education today has caused many educators — alternative and other- 
wise — to rush headlong into objectives as a source of stability and 
"penranence." Let^s agree on sometliing: much of what is important 
to the learning process cannot be adequately defined in behavioral 
—or other wise measurable-- terms. Given th-: assumption, one must 
recognize that objectives can only be viewed as a partial solution 
to the evaluation problem. Objectives appropriate for alternative 
programs need to be de\^eluped by individuals sensitive to both the 
needs and the processes of affective learning. Objectify where you 
can, but with full cognizance of its limitations. Is it not better 
that the program personnel set the program^s objectives than to have 
them imposed entiraly by an outside evaluator? 

6. A major problem in evaluating alternative programs is the lack of 
qualified evaluators who have tht sensitivities and insights neces- 
sary to fully understand the concept of alternative education and 
measure its implementation. One means of compensating for this lack 
is the development by the staff of well defined, specific objectives 
wherever possible, so that the evaluator is forced to evaluate the 
program on those terms. Any efforts the alternative program staff 
is able to make to reduce the feelings of hostility and suspicion 
which often develop toward evaluators will probably aid in reducing 
the possibility of irrelevant interpersonal factors prejudicing the 
interest of the program from the evaluator 's standpoint. 

7. A present, the majority of program evaluations conducted in the 
area of alternative education are imposed by some outside source, 
such as a categorical funding source, or the District administration. 
The evaluation is conducted to mt^et the needs of the District or the 
f'.inding source and to pro'dide information for decision making of 
various kinds. However, what is often overlooked is the program 
improvement aspect of evaluation and the internal needs of the 
program which should also be considered in any evaluation that is 
conducted. Obviously any program can be over evaluated and most 
measurement techniques in use today often involve intrusions into 
tb^ instructional program. By coordinating the evaluation and 
considering both the internal and external needs for evaluative data, 
such disruption can be reduced and the evaluation process can be 
more cognizant of the interrelated needs of the program and its 
supporting agency. 

8« The emphasis in evaluation, particularly in the era of behavioral 
objectives, has been on the product or outcome of the instructional 
process. Because many alternative education programs emphasize the 
process by which one takes place, new criteria for evaluation need 
to be developed which focus on that process. However, no program 
can be adequately evaluated solely on the basis of either product 
or of process criteria. 

9. A common fault of people inexperienced in the evaluation of alterna- 
tive programs is that they establish performance criteria on the 
basis of what the traditional program is supposed to be doing, whether 
Y:nfr- ^^^^ ^® ^ emphasis of the alternative program. Most alter- 

£jv^ native programs were developed to fill a need that was not being met 

by Che regular school program. It stands to reason, therefore, that 
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alternative programs must be evaluated on the basis of what they were 
designed to do. For example, a fifty percent attendance level on the 
part of students in the dropout program may be a very positive elfect 
of a given program. After all, the dropouts were previously totally 
out of school and therefore fifty percent attendance would be a 
marked improvement, 

10. One of the argumencs* for alternative education has traditionally 
been that ours is a pluralistic society in which there is no clear 
cut majority and the needs of the various elements of that society 
differ substantially. In the proces'i of evaluation this diversity 
becomes a problem in that education, including alternative programs, 
serves a variety of audiences. These range from the students and 
staff of the programs to the School Board, to the community, to 
various local ar.d state political leaders, etc. Obviously some* of 
these audiences are more directly irelated with the educational pro- 
gram and it is likely that each one of those may hold different 
expectations for the criteria of success within the same program. 
Tills is a dilemma which cannot readily be resolved, however aware- 
ness of the fact that such diverse expectacions may exist can perhaps 
make the program more responsive, at least in part, to them* 

11. A final problem is the unsubstantiated assumption that development 
in the cognitive domain of learning is neglected when a program 
emphasizes the effective and/or psychomotor domains in its instruc- 
tional efforts. Education has traditionally focused only on the cog- 
nitive domain and tht alternative education irovement reflects a 
reaction to this overemphasis. However, critics of alternative edu- 
cation have indicated that the need still exists for "basic education," 
which is their terminology for the rote learning processes which often 
accompany coguitive learning. Much recent research has indicated that 
the attitude and self-concep': of the learner has a profound effect on 
his receptivity to cognitive learning. While this factor incompasses 
more than simply evaluative concerns regarding alternative programs, 

it is an element which must be recognized in designing an evaluative 
process so as to avoid the tendency to measure only the more easily 
measurable cognitive elements. 

Evaluation Processes for Alternative Programs 

Evaluation within alternative programs cannot be separated from the planning 
process. It begins with an identification of needs ;md the establishment of 
goals and objectives based on those needs. By considering evaluation as part 
of the planning process, goals and objectives can Le considered from the 
perspective of the ease with which they can be evaluated. This is meant to 
imply only that some means of evaluation can be established for practically 
any program objective, particularly if the need for evaluative data is estab- 
lished early enough in the program so that appropriate measures can be built in. 

In order to maintain the integrity of alternative programs, it seems appropriate 
that the program staff be the primary source of need identification and the set- 
ting of goals and objectives for the program. However, because of the external 
evaluation needs (i.e., the central administration or the funding agency) the 
administrative unit responsible for the alternative program needs to review 
the goals and objectives, the evaluative procedure proposed by the program, 
and may then suggest additional objectives which were not considered, or pos- 
sible modifications of the program objectives which may lend themselves more 
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directly to evaJ.uatioa without diluting the intent of the program. As part 
of this process, the evaluative criteria are Jc^ttttiy^stablished by the pro- 
gram staff at the central administration keeping in mind the internal needs 
of the program, as well as those of the outside agency^ 

Although what is evaluated in a program can be determined only by the objec- 
tives of that program, the following are some elements which might be appro- 
priately evaluated in an alternative program. In each case j*: assumes that h 
the objective on which the item to be evaluated is based is self-ev.-ldent to 
the reader. 

1. Community attitude toward the program 

2. Staff attitude and attitude change over the course of the program 

3. Parental attitude 

4. Student attitude 

5. Community participation (i.t-., as volunteers, on field trips, visits 
to the school) 

6. Academic achievement (not for a success/failure judgment, but for diag- 
nostic information which may be of use in certain situations) 

7. Academic participation - this is differentiated froja achievement ia 
that it is intended to be descriptive data indicating how many students 
are taking what courses and for what extent of time 

8« Attendance data from both teachers and students 
9. Discipline and suspension figure 

10. The extent and nature of feedback to the community 

11. A follow up survey of program graduates 

12. The holding pov/er of the program 

13. Changes in student -family relationships 

14. Program development and growth, both in the size of the program 
and the scope of its offering 

15. Student activities outside the school: Community work, social work, 
service 

16. At the elementaty level, the number of kids who dash out of the room 
at recess (in some schools recess is the most exciting thing that 
happens 1) 

While some might argue that many of these are trivial bases for evaluation, 
each of them is in part an indicator of something that might be happening 
the program. The more specific evaluative data that can be developed and 
provided by the program itself to its cvitics and detractors, the more solid 
the image of the program becomes in the eyes of the public and even the regu- 
lar school administration. 

While one might very well dr^cry the lack of adequate affective measures for 
alternative programs, it should be recognized that the academic element of 
the program does need to be evaluated. Standard"^ zed tests, although much 
maligned (and with some justification) if carefully selected can be a use- 
ful measurement instrument, provided they are not the only indicator th^t 
is used. 

R ole of the Consortium 

The National Consortium for Alternatives in Education would seem to have a 
major role in the process of evaluation as it relates to alternative programs 
O , on the national level. The principal elements of this role for consideration 



EVALUATING EDUCATIONAL ALTERNATIVES 
Page 7 



by the consortium might include the following: 

!• To serve as a resource bank of evaluation instruments and processes 

2. To develop aad make available guidelines for the establishment of 
evaluation programs 

3. To serve as a depository of summary information or surveys of evalua- 
tion results from around the country 

4. To serve as a 'resource person bank for people with evaluation exper- 
tise in the area of alternative education 

5» To conduct regional workshops which partially or totally focus on 
the problems of evaluation in alternative education. 

These would seem to be appropriate and necessary roles for a body such as 
the consortium. Through this mechanism at the national level, local prograai 
efforts could be improved substantially and many of the problems which now 
exist in the area of alternative program evaluation could be significantly 
reduced • 

lu summary, as was previously stated, the age of accountability in education 
Is here. Educational programs, particularly alternative programs face the 
tJioice of developing evaluation programs so that their effect can be assessed 
on their own terms, or to be evaluated on the basis of terms imposed by an 
outside agency. The best defense is a good offense - particularly in the 
field of eltemative education. 



This paper is a synthesis of ideas discussed in Work Group III at the Wingspread 
Conference on Educational Alternatives, April, 1972, Racine, Wisconsin. 
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